Image processing device, photographing control method, and program

ABSTRACT

An information processing system that acquires first image data captured by a first camera unit disposed on a first side of a housing; acquires second image data captured by a second camera unit disposed on a second side of the housing, which is opposite to the first side of the housing; and modifies the second image data based on the first image data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation application of application Ser. No.14/760,323, filed Jul. 10, 2015 which is a national stage ofPCT/JP2014/000677, filed Feb. 7, 2014, which claims the benefit ofJapanese Patent Application JP 2013-031376 filed Feb. 20, 2013, thecontents of which are incorporated herein by reference in theirentirety.

TECHNICAL FIELD

The present technology relates to an image processing device, aphotographing control method, and a program, and more particularly, animage processing device, a photographing control method, and a programcapable of obtaining a higher-quality image.

BACKGROUND ART

In the past, imaging devices on which two cameras are mounted have beenknown. Some of the imaging devices can simultaneously photograph imageswith two cameras.

For example, as the imaging devices, imaging devices have been suggestedwhich performs control such that imaging is configured to be performedwith a main camera when a subject is attempted to be imaged with themain camera and an expression of a photographer imaged with a sub-camerais changed (for example, see PTL 1).

CITATION LIST Patent Literature [PTL 1] Japanese Unexamined PatentApplication Publication No. 2010-11253 SUMMARY Technical Problem

However, in the above-described technology, a photographing timing iscontrolled by the main camera based on an image obtained with thesub-camera. Thus, a higher-quality image such as an image having highersharpness or an image closer to a point of sight of the photographer maynot be obtained.

It is desirable to provide a technology capable of a higher-qualityimage.

Solution to Problem

According to one aspect, the disclosure is directed to an informationprocessing system including circuitry configured to acquire first imagedata captured by a first camera unit disposed on a first side of ahousing; acquire second image data captured by a second camera unitdisposed on a second side of the housing, which is opposite to the firstside of the housing; and modify the second image data based on the firstimage data.

The first image data may be captured by the first camera unit atsubstantially the same time as the second image data is captured by thesecond camera unit.

The circuitry may be configured to modify the second image data byperforming blur correction on the second image data based on the firstimage data.

The circuitry may be configured to: generate first motion informationbased on the first image data; generate second motion information basedon the second image data; and modify the second image data by performingblur correction on the second image data based on the first motioninformation and the second motion information.

The circuitry may be configured to modify the second image data bycontrolling an image capture angle corresponding to the second imagedata based on the first image data.

The circuitry maybe configured to control the image capture anglecorresponding to the second image data by at least one of changing azoom ratio of the second camera unit, changing a diaphragm of the secondcamera unit, changing an orientation of the second camera unit, andperforming image deformation of the second image data.

The circuitry maybe configured to: output the second image data to adisplay; detect a gaze position of a user on the display based on thefirst image data; and modify the second image data by controlling animage capture parameter of the second camera unit based on the detectedgaze position of the user on the display.

The circuitry may be configured modify the second image data bysynthesizing the first image data and the second image data.

The circuitry maybe configured to: obtain a first white balance valuecorresponding to the first image data; obtain a second white balancevalue corresponding to the second image data; and control a light sourceto emit light based on the first white balance value and the secondwhite balance value.

The circuitry may be configured to: acquire third image data captured bythe first camera unit while the light source is emitting light; andmodify the second image data by synthesizing the third image data andthe second image data.

According to another aspect, the disclosure is directed to aninformation processing method including: acquiring first image datacaptured by a first camera unit disposed on a first side of a housing;acquiring second image data captured by a second camera unit disposed ona second side of the housing, which is opposite to the first side of thehousing; and modifying the second image data based on the first imagedata.

According to another aspect, the disclosure is directed to anon-transitory computer-readable medium including computer-programinstructions, which when executed by an information processing system,cause the system to: acquire first image data captured by a first cameraunit disposed on a first side of a housing; acquire second image datacaptured by a second camera unit disposed on a second side of thehousing, which is opposite to the first side of the housing; and modifythe second image data based on the first image data.

Advantageous Effects of Invention

According to an embodiment of the present technology, it is possible toobtain a higher-quality image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of the outerappearance of an image processing device.

FIG. 2 is a diagram illustrating a configuration example of the imageprocessing device.

FIG. 3 is a diagram illustrating a configuration example of a signalprocessing unit.

FIG. 4 is a flowchart for describing a photographing process.

FIG. 5 is a diagram for describing motion information.

FIG. 6 is a diagram for describing the motion information.

FIG. 7 is a diagram for describing a camera blur correction.

FIG. 8 is a diagram illustrating a configuration example of an imageprocessing device.

FIG. 9 is a flowchart for describing a photographing process.

FIG. 10 is a diagram for describing field angle adjustment.

FIG. 11 is a diagram illustrating a configuration example of a signalprocessing unit.

FIG. 12 is a flowchart for describing a photographing process.

FIG. 13 is a diagram for describing field angle adjustment.

FIG. 14 is a diagram for describing focal point adjustment.

FIG. 15 is a diagram illustrating a configuration example of a signalprocessing unit.

FIG. 16 is a flowchart for describing a photographing process.

FIG. 17 is a diagram for describing light source matching.

FIG. 18 is a diagram illustrating a configuration example of a signalprocessing unit.

FIG. 19 is a flowchart for describing a photographing process.

FIG. 20 is a diagram illustrating a configuration example of the outerappearance of an image processing device.

FIG. 21 is a diagram illustrating a configuration example of an imageprocessing device.

FIG. 22 is a flowchart for describing a photographing process.

FIG. 23 is a flowchart for describing a photographing process.

FIG. 24 is a diagram illustrating a configuration example of a computer.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments to which the present technology is applied willbe described with reference to the drawings.

First Embodiment Overview of Embodiments of Present Technology

An embodiment of the present technology relates to devices including aplurality of imaging units, such as a multi-function type mobile phoneor a digital camera, which can photograph images in mutually differentdirections and performing photographing control based on informationobtained from an image photographed by another imaging unit when a mainimaging unit photographs an image.

The photographing control performed here is control for adjustmentrelevant to an image to be photographed, for example, driving control ofan imaging unit such as image view adjustment or illumination of asubject or image quality adjustment of an image such as camera blurcorrection or white balance adjustment. The photographing control isassumed to include control of image processing performed up to recordingof an image after photographing of the image to obtain a final image forrecording. In embodiments of the present technology, a higher-qualityimage can be obtained by performing such photographing control.

FIG. 1 is a diagram illustrating a configuration example of the outerappearance of an image processing device such as a multi-function mobilephone to which an embodiment of the present technology is applied.

In FIG. 1, the outer appearance of an image processing device 11indicated by an arrow A11 is a front surface side, that is, an outerappearance side viewed from a user when the user holds the imageprocessing device 11 to perform an operation. The outer appearance ofthe image processing device 11 indicated by an arrow A12 is a sidesurface side, that is, an outer appearance side when viewed from theleft side or the right side of the image processing device 11 indicatedby the arrow A11 in the drawing. The outer appearance of the imageprocessing device 11 indicated by an arrow A13 is a rear surface side,that is, an outer appearance of an opposite surface side to the frontsurface side.

As indicated by the arrow A11, a display unit 21 that displays an imageor the like is installed on the front surface side of the imageprocessing device 11. A touch panel used for the user to perform aninput operation is installed to be superimposed on the display unit 21.In the drawing of the display unit 21 on the front surface of the imageprocessing device 11, an in-image capturing unit 22 that photographs asubject present in the direction of the front surface side of the imageprocessing device 11 is also installed on the upper side. Accordingly,an image photographed by the in-image capturing unit 22 is an image ofthe user operating the image processing device 11 in most cases.Hereinafter, an image photographed by the in-image capturing unit 22 isreferred to as an in-image. The description will be continued assumingthat an image of a user is photographed as the in-image.

As indicated by the arrow A13, an out-image capturing unit 23 thatphotographs a subject present in the direction of the rear surface sideof the image processing device 11 is installed on the rear surface ofthe image processing device 11. In this example, the out-image capturingunit 23 is installed to photograph an image in an opposite direction tothe photographing direction of the in-image capturing unit 22. However,the in-image capturing unit 22 and the out-image capturing unit 23 maybe disposed such that photographing directions are different from eachother.

An image photographed by the out-image capturing unit 23 is normally animage of a landscape, a human, or the like viewed by the user, that is,an image of a subject which the user considers desirable to photograph.Hereinafter, an image photographed by the out-image capturing unit 23 isreferred to as an out-image.

The image processing device 11 photographs an in-image together with anout-image at the time of photography of the out-image and performsphotographing control to obtain the out-image with higher sharpness byremoving a camera shake component of the out-image using the in-image.That is, camera blur correction is performed as the photographingcontrol of the out-image. In this embodiment, moving images are assumedto be photographed as the out-image and the in-image.

Configuration Example 1 of Image Processing Device

Next, a more detailed configuration of the image processing device 11will be described.

FIG. 2 is a block diagram illustrating a more detailed configurationexample of the image processing device 11. In FIG. 2, the same referencenumerals are given to constituent elements corresponding to theconstituent elements in FIG. 1 and the description thereof will beappropriately omitted.

The image processing device 11 illustrated in FIG. 2 includes thedisplay unit 21, the in-image capturing unit 22, the out-image capturingunit 23, an operation input unit 51, a control unit 52, a signalprocessing unit 53, and a recording unit 54.

For example, the operation input unit 51 is formed by the touch panelsuperimposed on the display unit 21 and supplies a signal according to auser's operation to the control unit 52.

The control unit 52 controls a process of the entire image processingdevice 11 according to the signal supplied from the operation input unit51. For example, the control unit 52 instructs the in-image capturingunit 22 or the out-image capturing unit 23 to photograph an image orinstructs the signal processing unit 53 to perform various kinds ofimage processing on the out-image. The control unit 52 instructs thedisplay unit 21 to display an image or instructs the recording unit 54to record an image.

The signal processing unit 53 performs camera blur correction on theout-image based on the out-image supplied from the out-image capturingunit 23 and the in-image supplied from the in-image capturing unit 22and supplies the out-image obtained as the result of the imageprocessing to the recording unit 54 to record the out-image. The signalprocessing unit 53 reads the out-image from the recording unit 54 andsupplies the out-image to the display unit 21 to display the out-image.

The recording unit 54 records the out-image supplied from the signalprocessing unit 53 and supplies the out-image recorded as necessary tothe signal processing unit 53.

Configuration Example 1 of Signal Processing Unit

More specifically, the signal processing unit 53 in FIG. 2 is configuredas in FIG. 3.

The signal processing unit 53 illustrated in FIG. 3 includes a motionestimation unit 81, a motion estimation unit 82, a synthesis unit 83,and a camera blur correction unit 84.

The motion estimation unit 81 generates motion information indicating amotion of the image processing device 11 at the time of the photographyof the out-image by performing motion estimation on the out-imagesupplied from the out-image capturing unit 23 and supplies the generatedmotion information to the synthesis unit 83.

The motion estimation unit 82 generates motion information indicating amotion of the image processing device 11 at the time of photography ofthe in-image by performing motion estimation on the in-image suppliedfrom the in-image capturing unit 22 and supplies the generated motioninformation to the synthesis unit 83. The in-image is simultaneouslyphotographed with the out-image.

The synthesis unit 83 obtains motion information indicating the finalmotion of the image processing device 11 at the time of the photographyof the out-image by synthesizing the motion information supplied fromthe motion estimation unit 81 and the motion information supplied fromthe motion estimation unit 82 and supplies the obtained motioninformation to the camera blur correction unit 84.

The camera blur correction unit 84 performs camera blur correction onthe out-image supplied from the out-image capturing unit 23 based on themotion information supplied from the synthesis unit 83 and supplies theout-image acquired as the result to the recording unit 54.

[Description 1 of Photographing Process]

Here, when the user operates the operation input unit 51, a mode forperforming the camera blur correction of the out-image is selected, andan instruction to photograph the out-image is given by simultaneousphotography of the out-image and the in-image, the image processingdevice 11 performs a photographing process to photograph the out-image.

Hereinafter, the photographing process performed by the image processingdevice 11 will be described with reference to the flowchart of FIG. 4.

In step S11, the out-image capturing unit 23 photographs an out-image inresponse to an instruction of the control unit 52 and supplies theout-image to the motion estimation unit 81 and the camera blurcorrection unit 84. For example, a moving image such as a landscapeimage is photographed as the out-image. At the time of the photographyof the out-image, the signal processing unit 53 supplies the out-imagesupplied from the out-image capturing unit 23 to the display unit 21 sothat the out-image is displayed. Thus, the user can perform thephotographing while confirming the out-image on the display unit 21, asnecessary.

In step S12, the in-image capturing unit 22 photographs an in-image inresponse to an instruction of the control unit 52 and supplies thein-image to the motion estimation unit 82. For example, a moving imageof the face of the user operating the image processing device 11 isphotographed as the in-image.

In step S13, the motion estimation unit 81 generates motion informationof the image processing device 11 by performing the motion estimation onthe out-image supplied from the out-image capturing unit 23 and suppliesthe motion information to the synthesis unit 83.

In step S14, the motion estimation unit 82 generates motion informationof the image processing device 11 by performing the motion estimation onthe in-image supplied from the in-image capturing unit 22 and suppliesthe motion information to the synthesis unit 83.

In step S15, the synthesis unit 83 synthesizes the motion informationsupplied from the motion estimation unit 81 and the motion informationsupplied from the motion estimation unit 82 and supplies the synthesizedmotion information to the camera blur correction unit 84.

Specifically, for example, the following processes are performed in stepS13 to step S15 described above.

That is, as indicated by an arrow A21 of FIG. 5, a group of a pluralityof images, that is, a moving image including images of respective framescan be assumed to be obtained as an out-image. Here, each rectangleindicated by the arrow A21 represents each frame of the out-image.

When the out-image including the plurality of frames is obtained, themotion estimation unit 81 divides the out-image into a plurality oflocal regions and obtains a motion vector of each local region, asindicated by an arrow A22. One rectangle indicated by the arrow A22represents an out-image and each rectangle in the out-image representsthe local region on the out-image. Further, an arrow drawn in each localregion indicates a motion vector of the local region.

In the calculation of the motion vector of the local region, forexample, a method of obtaining an amount of movement of a subject bysimilarity between images at respective times (frames), for example, atemplate matching method, is used. Thus, when the motion vector isobtained for each local region of the out-image, the motion estimationunit 81 calculates motion vector A of the image processing device 11 asmotion information based on the motion vector of each local region ofthe out-image at each time.

For example, motion vector A of the image processing device 11 iscalculated according to a method such as affine parameter estimation bya Newton method.

As in the case of the out-image, the motion information based on thein-image is also calculated. That is, when a group of images indicatedby an arrow A23 is obtained as an in-image, the motion estimation unit82 divides the in-image into a plurality of local regions, as indicatedby an arrow A24, and obtains a motion vector of each local region bytemplate matching or the like.

Each rectangle indicated by the arrow A23 represents each frame of thein-image. One rectangle indicated by the arrow A24 represents anin-image and each rectangle in the in-image represents the local regionon the in-image. Further, an arrow drawn in each local region indicatesa motion vector of the local region.

The motion estimation unit 82 calculates motion vector B of the imageprocessing device 11 as motion information by a Newton method or thelike based on the motion vector of each local region of the in-image ateach time.

The right, left, upper, and lower portions of motion vector B obtainedfrom the in-image in this way are reversed with respect to motion vectorA obtained from the out-image, as illustrated in FIG. 6.

That is, as illustrated in FIG. 6, a direction perpendicular to thefront surface of the image processing device 11, that is, a directionparallel to the photographing direction of the out-image capturing unit23 is referred to as a Z axis direction and axes perpendicular to the Zaxis are referred to as an X axis and a Y axis. Here, the X axis is anaxis in the horizontal direction of the image processing device 11, thatis, the direction parallel to the horizon plane and the Y axis is anaxis in the direction parallel to the vertical direction.

When motion vector A and motion vector B are calculated using theout-image and the in-image, respectively, and for example, a motion ofthe image processing device 11 is only a rotation motion around the Yaxis serving as a rotation axis, as illustrated in the upper side of thedrawing, motion vector A and motion vector B are in the same direction.

On the other hand, when the motion of the image processing device 11 isonly a rotation motion around the X axis serving as a rotation axis anda rotation motion around the Z axis serving as a rotation axis, asillustrated in the lower side of the drawing, the right, left, upper,and lower portions of motion vector A and motion vector B are reversedso that the directions thereof are reversed to be opposite directions.

One rectangle in the right side of the drawing represents the out-imageor the in-image and an arrow in the image represents a motion vector ofthe image.

When motion vector A and motion vector B are obtained as the motioninformation in this way, the synthesis unit 83 synthesizes motion vectorA and motion vector B to generate motion vector C. The obtained motionvector C is assumed to be the final motion information of the imageprocessing device 11.

Specifically, for example, the synthesis unit 83 calculates the motionvector C by calculating the following Expression (1).

[Math. 1]

C=a1·A·b1·B  (1)

In Expression (1), a1 and b1 indicate coefficients which are weights formotion vector A and motion vector B, respectively, and are determined sothat “a1+b1=1.0” is satisfied.

For example, the out-image capturing unit 23 and the in-image capturingunit 22 may perform the photographing so that exposure times differ anda larger weight may be given to a motion vector obtained from an imagefor which the exposure time is shorter, so that the coefficients a1 andb1 are determined according to the accuracy of the motion estimation.Here, the reason why the larger weight is given to the motion vectorobtained from the image for which the exposure time is shorter is thatthe image for which the exposure time is shorter at the time of thephotography of the image has less blurring caused due to a movingsubject or camera shake and the motion estimation can be performedaccurately.

In particular, of the out-image capturing unit 23 and the in-imagecapturing unit 22, a distance up to a subject is shorter for thein-image capturing unit 22 photographing a user who is a photographer inmany cases. Therefore, blurring (shake) caused due to camera shake isless in the in-image than in the out-image and the motion estimation canbe performed with higher accuracy. Accordingly, more effectiveness canbe acquired when the exposure time of the in-image is set to be shorterthan the exposure time of the out-image and the coefficient b1 is set tohave a larger value than the coefficient a1.

Referring back to the flowchart of FIG. 4 to make the description, whenthe motion information of the image processing device 11 is calculatedas the control information for camera blur correction in the process ofstep S15, the process of step S16 is subsequently performed.

In step S16, the camera blur correction unit 84 performs the camera blurcorrection on the out-image supplied from the out-image capturing unit23 based on the motion information supplied from the synthesis unit 83and supplies the out-image obtained as the result to the recording unit54.

For example, the camera blur correction unit 84 calculates an amount ofmovement of a feature point inside each local region on the out-imagebased on the motion vector C which is the motion information suppliedfrom the synthesis unit 83 and calculates an affine parameter based onthe amount of movement.

More specifically, as illustrated in the upper side of FIG. 7, a motionvector of each local region is assumed to be obtained as motioninformation of the image processing device 11 in an out-image AP11 at atime t (frame t). Each rectangle in the out-image AP11 represents thelocal region and an arrow in each local region represents a motionvector of the local region.

Now, as illustrated in the lower side of the drawing, feature points areassumed to be obtained for each local region of the out-image AP11. Inthe lower side of the drawing, one circle, more specifically, a circlewith no diagonal represents one feature point at a time t and a circlewith diagonals represents a feature point at a time t+1.

In this case, the camera blur correction unit 84 obtains the motionvector C as motion information, that is, the motion vector of each localregion, and the position of the feature point at the time t+1 based onthe position of each feature point at the time t. Then, the camera blurcorrection unit 84 obtains deformation parameters a to 1 satisfying thefollowing Expression (2) and Expression (3) as affine parameters. Here,P feature points are assumed to be on the out-image AP11.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{{\underset{a,b,c,d,e,f,g,h,i,j,k,l}{argmin}{\sum\limits_{p}\; \begin{bmatrix}x \\y \\z\end{bmatrix}}} - \begin{bmatrix}s \\t \\u\end{bmatrix}} & (2) \\\left\lbrack {{Math}\mspace{14mu} 3} \right\rbrack & \; \\{\begin{bmatrix}x \\y \\z\end{bmatrix} = {{\begin{bmatrix}a & b & c \\d & e & f \\g & h & i\end{bmatrix}\begin{bmatrix}X \\Y \\Z\end{bmatrix}} + \begin{bmatrix}j \\k \\l\end{bmatrix}}} & (3)\end{matrix}$

In Expression (2), a vector having elements s, t, and u indicates thecoordinates of the feature points of the out-image AP11 at the time t.That is, the vector having the elements s, t, and u indicates thecoordinates of the feature points represented by the circles with nodiagonal in FIG. 7.

In Expression (3), a vector having elements X, Y, and Z indicates thecoordinates of the feature points of the out-image AP11 at the time t+1obtained from the motion information. That is, the vector havingelements X, Y, and Z indicates the coordinates of the feature pointsrepresented by the circles with the diagonals in FIG. 7.

Expression (3) indicates image deformation in which the feature pointsat the time t+1 in FIG. 7 overlap the feature points at the time t, thatis, a deformation expression of the affine transform. Accordingly, inExpression (2) and Expression (3), a vector having elements x, y, and zindicates the coordinates of the feature points after the deformationwhen the image deformation is performed on the coordinates of thefeature points of the out-image AP11 at the time t+1 using the affineparameters.

When the deformation parameters a to l are calculated, as indicated inExpression (2), the deformation parameters a to l are obtained for the Pfeature points matched with each other so that differences between thepositions of the feature points at the time t+1 after the deformationand the positions of the feature points at the time t, that is, atransform error is the minimum.

When the affine parameters are obtained in this way, the camera blurcorrection unit 84 performs motion correction by performing imagedeformation indicated in the foregoing Expression (3) using the obtainedaffine parameters and deforming the actual out-image at the time t+1supplied from the out-image capturing unit 23. That is, the camera shakecomponent is removed from the out-image, the camera blur correction isperformed, and the out-image obtained as the result is supplied to therecording unit 54. In the camera blur correction by the imagedeformation (image transform) indicated in Expression (3), motions oftranslation, expansion and contraction, rotation, and distortion can betreated with.

Referring back to the flowchart of FIG. 4 to make the description, whenthe camera blur correction is performed on the out-image in step S16,the process proceeds to step S17.

In step S17, the recording unit 54 records the out-image supplied fromthe camera blur correction unit 84 under the control of the control unit52, and then the photographing process ends.

As described above, the image processing device 11 performs the camerablur correction on the out-image based on the motion informationacquired by obtaining the motion information of the image processingdevice 11 using the out-image and the in-image.

Thus, the motion of the image processing device 11 can be estimated withhigher accuracy using both of the out-image and the in-image. Thus, evenwhen the image processing device 11 moves more complexly at the time ofthe photography of the out-image, a shake component occurring in themotion of the image processing device 11 can be removed with higheraccuracy. That is, it is possible to obtain the out-image with highquality and higher sharpness.

In other words, in the image processing device 11, by performing thephotographing control on the images using information regarding theimages obtained through the photographing in two different directions,the photographing control can be performed using more pieces ofinformation compared to a case in which one image is photographed. Thus,it is possible to obtain the out-image with higher quality.

The example in which the camera blur correction is performed on theout-image has been described above. However, the camera blur correctionmay be performed on the in-image through the same process and thein-image obtained as the result may be recorded.

Modification Example 1 of First Embodiment Configuration Example 2 ofImage Processing Device

As described above, the motion vector C has been obtained bysynthesizing the motion vector A obtained from the out-image and themotion vector B obtained from the in-image. However, a motion vectorobtained by a sensor may be also synthesized in addition to the motionvectors.

In this case, the image processing device 11 has, for example, aconfiguration illustrated in FIG. 8. In FIG. 8, the same referencenumerals are given to constituent elements corresponding to theconstituent elements illustrated in FIG. 2 and the description thereofwill be appropriately omitted.

The image processing device 11 illustrated in FIG. 8 is configured suchthat a motion detection unit 111 is further installed in the imageprocessing device 11 in FIG. 2.

The motion detection unit 111 includes, for example, an accelerationsensor or a gyro sensor, detects a motion of the image processing device11, and supplies a motion vector D indicating the detection result asmotion information to the synthesis unit 83 of the signal processingunit 53.

Here, the signal processing unit 53 of the image processing device 11 inFIG. 8 has the configuration of the signal processing unit 53illustrated in FIG. 3. The synthesis unit 83 of the signal processingunit 53 synthesizes motion information supplied from the motionestimation unit 81, motion information supplied from the motionestimation unit 82, and the motion information supplied from the motiondetection unit 111 and supplies the synthesized motion information tothe camera blur correction unit 84.

[Description 2 of Photographing Process]

Next, a photographing process performed when the image processing device11 has the configuration illustrated in FIG. 8 will be described withreference to the flowchart of FIG. 9. Since processes of step S41 andstep S42 are the same as the processes of step S11 and step S12 of FIG.4, the description thereof will be omitted.

In step S43, the motion detection unit 111 detects a motion of the imageprocessing device 11 and supplies the motion vector D indicating thedetection result as the motion information to the synthesis unit 83.

When the motion vector D is calculated, processes of step S44 and stepS45 are subsequently performed to perform motion estimation on anout-image and an in-image. Since these processes are the same as theprocesses of step S13 and step S14 of FIG. 4, the description thereofwill be omitted.

In step S46, the synthesis unit 83 synthesizes the motion informationsupplied from the motion estimation unit 81, the motion informationsupplied from the motion estimation unit 82, and the motion informationsupplied from the motion detection unit 111 and supplies the synthesizedmotion information to the camera blur correction unit 84. Specifically,for example, the synthesis unit 83 calculates the motion vector C bycalculating the following Expression (4).

[Math. 4]

C=a1·A+b1·B+c1·D  (4)

In Expression (4), c1 indicates a coefficient serving as a weight forthe motion vector D and coefficients a1, b1, and c1 are determined sothat “a1+b1+c1=1.0” is satisfied. Accordingly, in Expression (4),weighted addition is performed on the motion vector A, the motion vectorB, and the motion vector D as the motion information, so that the motionvector C which is the final motion information of the image processingdevice 11 is obtained.

Thus, when the motion information of the image processing device 11 isobtained, processes of step S47 and step S48 are subsequently performedand the photographing process ends. Since these processes are the sameas the process of step S16 and step S17 of FIG. 4, the descriptionthereof will be omitted.

Thus, the image processing device 11 synthesizes the motion informationobtained from the out-image and the in-image and the motion informationobtained by the motion detection unit 111 and performs the camera blurcorrection on the out-image based on the motion information obtained asthe result.

Accordingly, by using not only the motion information obtained from theout-image and the in-image but also the motion information obtained bythe motion detection unit 111, it is possible to estimate the motion ofthe image processing device 11 with higher accuracy. Thus, since theshake component can be removed through the camera blur correction withhigher accuracy, it is possible to obtain the out-image with higherquality.

Second Embodiment Configuration Example 2 of Signal Processing Unit

The example in which the camera blur correction of the out-image isperformed as the photographing control has been described above.However, adjustment of a field angle of an out-image may be performed asphotographing control.

In this case, for example, as illustrated in FIG. 10, adjustment of afield angle of an out-image is performed according to a sight linedirection of user who is a photographer.

Specifically, as indicated by an arrow A31, normally, when a user U11images target subjects by orienting the out-image capturing unit 23 ofthe image processing device 11 in directions in which a subject H11 anda subject H12 are located, the user U11 performs the photographing,viewing images displayed on the display unit 21. Accordingly, since theposition of the out-image capturing unit 23 photographing the out-imageis actually different from a viewpoint position PE11 of the user U11viewing the subject, an image intended by the user U11 may notnecessarily be obtained.

In this example, straight lines AG11 indicate a field angle of theout-image capturing unit 23, that is, a photographing range, and twosubjects H11 and H12 are contained within a region in the photographingrange. For this reason, as indicated by an arrow A32, the subjects H11and H12 are shown in the out-image photographed by the out-imagecapturing unit 23. Since the out-image is displayed on the display unit21, the user U11 can perform photographing, viewing the display unit 21.

In practice, however, the user U11 may not desire to photograph both ofthe subjects H11 and H12. In this example, as indicated in the upperside of the drawing, when the user U11 views the subject H11 on thescreen of the display unit 21, there is a high probability that the userU11 pays attention to the subject H11. In FIG. 10, dotted lines AG12indicate a sight line of the user U11, that is, the range of a viewingfield, and thus the sight line of the user U11 is oriented in thedirection of the subject H11.

Thus, the image processing device 11 performs field angle control of theout-image capturing unit 23 as the photographing control of theout-image. For example, the image processing device 11 adjusts the fieldangle of the out-image capturing unit 23 so that the photographing rangeof the out-image capturing unit 23 is the range indicated by thestraight lines AG13.

The adjustment of the field angle is realized, for example, by changinga zoom magnification or a diaphragm of the out-image capturing unit 23,inclining the out-image capturing unit 23, more specifically, an imagingunit in the out-image capturing unit 23, or performing image deformationon the out-image.

When the adjustment of the field angle of the out-image capturing unit23, that is, the adjustment of the field angle of the out-image isperformed, as indicated by an arrow A33, only the subject H11 iscontained in the out-image. That is, the photographing range of theout-image capturing unit 23 becomes close to the range of the viewingfield viewed from the viewpoint position PE11 of the user U11. Thus, animage shown near the viewpoint of the user U11 and close to an imageintended by the user U11, that is, a higher-quality image can beobtained.

When the adjustment of the field angle of the out-image is performed asthe photographing control, the signal processing unit 53 of the imageprocessing device 11 illustrated in FIG. 2 has a configurationillustrated in FIG. 11.

The signal processing unit 53 illustrated in FIG. 11 includes a positioncalculation unit 141 and a field angle transformation processing unit142.

The position calculation unit 141 calculates a viewpoint position of theuser, that is, a photographer who operates the image processing device11 based on the in-image supplied from the in-image capturing unit 22and supplies the viewpoint position to the field angle transformationprocessing unit 142.

The field angle transformation processing unit 142 calculatesinformation necessary for the adjustment of the field angle of theout-image based on the viewpoint position supplied from the positioncalculation unit 141 and supplies the information to the control unit52. The control unit 52 generates control information for controllingthe out-image capturing unit 23 based on the information from the fieldangle transformation processing unit 142. Then, the control unit 52performs the adjustment of the field angle by changing the zoommagnification or the diaphragm of the out-image capturing unit 23 orinclining the imaging unit of the out-image capturing unit 23 based onthe control information.

The field angle transformation processing unit 142 includes an imagedeformation unit 151. The image deformation unit 151 transforms thefield angle of the out-image by performing the image deformation on theout-image from the out-image capturing unit 23 according to theviewpoint position from the position calculation unit 141 and suppliesthe changed field angle to the display unit 21 and the recording unit54.

[Description 3 of Photographing Process]

Next, a photographing process performed by the image processing device11 will be described with reference to the flowchart of FIG. 12 when thesignal processing unit 53 has the configuration illustrated in FIG. 11.

The photographing process is performed when a mode in which thephotographing is performed while the adjustment of the field angle ofthe out-image is performed is selected by the user and an instruction tostart photographing the out-image is given.

Since processes of step S71 and step S72 are the same as the processesof step S11 and step S12 of FIG. 4, the description thereof will beomitted. However, an out-image photographed in step S71 is supplied tothe field angle transformation processing unit 142 and an in-imagephotographed in step S72 is supplied to the position calculation unit141.

In step S73, the position calculation unit 141 calculates the viewpointposition of the user based on the in-image supplied from the in-imagecapturing unit 22.

For example, the position calculation unit 141 obtains the viewpointposition by detecting the face or eyes of the user from the in-image andcalculating the position of the eyes of the user from the in-imagecapturing unit 22, that is, the distance up to the viewpoint position,the height of the viewpoint position, and an amount of deviation betweenthe right and left of the viewpoint position with respect to thein-image capturing unit 22. The viewpoint position obtained in this wayis a 3-dimensional position, that is, the position of the viewpoint ofthe user in a 3-dimensional coordinate system in which a predeterminedposition is set as a reference. Further, the sight line direction of theuser may be obtained from the position of the pupils of the user in thein-image.

In step S74, the control unit 52 generates the control information forchanging the field angle of the out-image capturing unit 23 based on theinformation supplied from the field angle transformation processing unit142.

In step S75, the control unit 52 performs the adjustment of the fieldangle of the out-image by controlling the out-image capturing unit 23based on the generated control information. Specifically, the controlunit 52 performs the adjustment of the field angle by changing the zoommagnification or the diaphragm of the out-image capturing unit 23 orinclining the imaging unit of the out-image capturing unit 23.

Thus, the out-image subjected to the adjustment of the field angle tosome extent is supplied from the out-image capturing unit 23 to thefield angle transformation processing unit 142.

In step S76, the image deformation unit 151 adjusts the field angle ofthe out-image by performing the image deformation on the out-imagesupplied from the out-image capturing unit 23 based on the viewpointposition supplied from the position calculation unit 141 and suppliesthe adjusted field angle to the display unit 21 and the recording unit54.

Here, the processes of step S74 to step S76 described above will bedescribed in more detail.

For example, as illustrated in the upper side of FIG. 13, when the userU11 who is a photographer normally views the center of a display screenof the display unit 21 at the time of the photography of an out-image,the sight line direction of the user U11 is obtained based on theviewpoint position PE11 of the user U11 and the position of the displayscreen of the display unit 21. In the upper side of FIG. 13, the sightline of the user U11 is indicated by a dotted line bound by theviewpoint position PE11 and the image processing device 11. In FIG. 13,the same reference numerals are given to the portions corresponding tothe portions in FIG. 10, and the description thereof will beappropriately omitted.

The field angle transformation processing unit 142 obtains the sightline direction of the user U11 based on the viewpoint position PE11 andthe position of the display unit 21 in step S74. Further, the fieldangle transformation processing unit 142 determines which out-image canbe acquired when the out-image capturing unit 23 is virtually moved tothe viewpoint position PE11 based on the obtained sight line directionand the viewpoint position PE11.

Now, as indicated by an arrow Q11 in the upper side of the drawing, theout-image capturing unit 23 is assumed to be virtually moved to theviewpoint position PE11 of the user U11. In this example, an amount ofmovement of the out-image capturing unit 23 to the viewpoint positionPE11, an amount of rotation of the out-image capturing unit 23, or thelike can be calculated based on the position of the out-image capturingunit 23, the viewpoint position PE11, and the sight line direction ofthe user U11.

Therefore, the field angle transformation processing unit 142 uniquelydetermines to which position a point on a predetermined 3-dimensionalspace is projected on the display screen of the display unit 21 beforethe movement of the out-image capturing unit 23 and to which positionthe point is projected on the display screen of the display unit 21after the movement of the out-image capturing unit 23.

Thus, the field angle transformation processing unit 142 obtains the3-dimensional positions of three or more subjects, that is, positions onthe 3-dimensional space through the estimation from the out-image andcalculates affine parameters between two images shown in the foregoingExpression (3) based on the obtained 3-dimensional positions of thethree or more points.

A technology for obtaining the positions before and after the movementof the projected point is described in, for example, “Multiple ViewGeometry in Computer Vision,” by Richard Hartley and Andrew Zisserman inCambridge University Press, March 2004.

Thus, when the affine parameters for transforming the out-image can beobtained in the image obtained at the time of the movement of theout-image capturing unit 23 to the viewpoint position PE11, theadjustment of the field angle is performed based on the affineparameters.

For example, the field angle transformation processing unit 142 suppliesthe obtained affine parameters to the control unit 52. Then, the controlunit 52 generates the field angle of the out-image capturing unit 23,that is, the control information for changing the field angle of the outimage based on the affine parameters from the field angle transformationprocessing unit 142. Here, the control information is, for example,information indicating an amount of transformation of the zoommagnification or the diaphragm of the out-image capturing unit 23 or aninclination angle of the imaging unit of the out-image capturing unit23. The control unit 52 controls the out-image capturing unit 23 basedon the control information and then performs the adjustment of the fieldangle.

The image deformation unit 151 performs the image deformation on theout-image based on the affine parameters to adjust the field angle ofthe out-image. That is, the image deformation unit 151 adjusts the fieldangle by performing signal processing (deformation process) on theout-image.

Here, translation or rotation components can be obtained in the affineparameters. Accordingly, for example, the zoom may correspond totranslation in the Z axis direction illustrated in FIG. 6, theinclination of the imaging unit may correspond to rotation in the X axisdirection illustrated in FIG. 6, and the image deformation maycorrespond to other adjustment of the field angle. In the rotation inthe X axis direction, the imaging unit may be inclined by rotating theimaging unit using the X axis as a rotation axis.

Through the adjustment of the field angle, as illustrated in the lowerside of FIG. 13, the out-image for which the out-image capturing unit 23seems to be located at the viewpoint position PE11 of the user U11 whois the photographer can be acquired.

In this example, an out-image AP21 indicates an out-image photographedby the out-image capturing unit 23 before the adjustment of the fieldangle. In the out-image AP21, a point HP11 on the subject H11 isprojected to the middle lower side in the out-image AP21.

On the other hand, an out-image AP22 indicates an out-image which isphotographed by the out-image capturing unit 23 after the adjustment ofthe field angle and is subjected to the image deformation by the imagedeformation unit 151. In the out image AP22, the point HP11 on thesubject H11 is projected to the middle upper side in the out-image AP22.

Thus, when the adjustment of the field angle is performed on theout-image, the out-image for which the user U11 seems to view the actualsubject H11 through the display unit 21, that is, the high-qualityout-image closer to the viewpoint of the user U11 can be obtained.

Referring back to the flowchart of FIG. 12 to make the description, theprocess proceeds from step S76 to step S77 when the adjustment of thefield angle on the out-image is performed and the out-image obtained asthe result is supplied to the display unit 21 and the recording unit 54.

In step S77, the display unit 21 displays the out-image supplied fromthe field angle transformation processing unit 142 and obtained afterthe adjustment of the field angle.

In step S78, the recording unit 54 records the out-image supplied fromthe field angle transformation processing unit 142 and obtained afterthe adjustment of the field angle, and then the photographing processends.

As described above, the image processing device 11 obtains the viewpointposition of the user based on the in-image and performs the adjustmentof the field angle of the out-image based on the viewpoint position.Thus, by performing the adjustment of the field angle on the out-image,the out-image according to the sight line of the user can be obtained.That is, it is possible to acquire the higher-quality out-image closerto the viewpoint of the user.

As described above, the example has been described in which both of theadjustment of the field angle by the out-image capturing unit 23 and theadjustment of the field angle by the image deformation of the imagedeformation unit 151, such as the adjustment of the zoom magnificationor the diaphragm, are performed. However, only one of the adjustments ofthe field angle may be performed.

Third Embodiment Configuration Example 3 of Signal Processing Unit

Control may be performed as photographing control such that a sight linedirection of the user is detected from an in-image and a focal point,lightness, white balance, or the like is adapted in correspondence witha gaze position of a user on an out-image obtained by the sight linedirection.

In this case, for example, as indicated by an arrow A41 of FIG. 14, theimage processing device 11 photographs an image of the user U11 who is aphotographer as an in-image and obtains a viewpoint position PE11 of theuser U11 based on the in-image. In FIG. 14, the same reference numeralsare given to constituent elements corresponding to the constituentelements illustrated in FIG. 10 and the description thereof will beappropriately omitted.

In this example, the user U11 orients the out-image capturing unit 23 totwo subjects H11 and H12 and photographs an out-image. At this time, animage received by the out-image capturing unit 23 is displayed as apreview image of the out-image on the display unit 21 installed on thesurface (front surface) of the image processing device 11 on the side ofthe user U11.

The user U11 gives an instruction to perform photographing, viewing thepreview image displayed on the display unit 21 and causes the out-imageto be photographed. In FIG. 14, a dotted arrow oriented from theviewpoint position PE11 to the image processing device 11 indicates asight line of the user U11.

When the image processing device 11 obtains the viewpoint position PE11from the in-image, the image processing device 11 obtains the sight linedirection of the user U11 based on the viewpoint position PE11 andspecifies a gaze position of the user U11 on the out-image from thesight line direction, as indicated by an arrow A42.

In the image processing device 11 indicated by the arrow A42, theout-image, more particularly, the preview image of the out-image isdisplayed on the display unit 21 and the position indicated by a starmark on the subject H11 is assumed to be the gaze position of the userU11. The gaze position is a position at which the sight line of the userU11 and the display screen of the display unit 21 intersect each otherand is a position of a region on the out-image which the user U11 paysattention to.

When the gaze position of the user U11 on the out-image is obtained, theimage processing device 11 controls the out-image capturing unit 23 asthe photographing control such that the subject H11 located at the gazeposition on the out-image is focused. The image processing device 11performs lightness adjustment or white balance adjustment so that thelightness (exposure) and white balance of the subject H11 located at thegaze position on the out-image are appropriate.

Thus, as illustrated in the lower right side of the drawing, the subjectH11 is focused on the out-image and the lightness or the white balanceof the subject H11 is appropriate. That is, it is possible to acquirethe higher-quality out-image subjected to the focusing, the lightnessadjustment, and the white balance adjustment by setting the subjectwhich the user U11 pays attention to as a reference.

Thus, when the focusing, the lightness adjustment, and the like areperformed setting a gaze region of the user as a reference, the signalprocessing unit 53 of the image processing device 11 illustrated in FIG.2 has, for example, a configuration illustrated in FIG. 15. The signalprocessing unit 53 illustrated in FIG. 15 includes a sight linedirection calculation unit 181 and a gaze position calculation unit 182.

The sight line direction calculation unit 181 calculates a sight linedirection of the user operating the image processing device 11 based onthe in-image supplied from the in-image capturing unit 22 and suppliesthe sight line direction to the gaze position calculation unit 182.

The gaze position calculation unit 182 obtains the gaze position basedon the sight line direction supplied from the sight line directioncalculation unit 181 and supplies the gaze position to the control unit52. The control unit 52 generates control information based on the gazeposition from the gaze position calculation unit 182 so that the subjectat the gaze position on the out-image is focused and the exposure(lightness) and the white balance of the subject at the gaze positionare appropriate. Then, the control unit 52 controls the out-imagecapturing unit 23 based on the generated control information.

[Description 4 of Photographing Process]

Next, a photographing process performed when the signal processing unit53 has the configuration illustrated in FIG. 15 will be described withreference to the flowchart of FIG. 16. Since processes of step S101 andstep S102 are the same as the processes of step S11 and step S12 of FIG.4, the description thereof will be omitted.

In step S103, the sight line direction calculation unit 181 calculatesthe sight line direction of the user who is a photographer based on thein-image supplied from the in-image capturing unit 22 and supplies thesight line direction to the gaze position calculation unit 182. Forexample, the sight line direction calculation unit 181 obtains theviewpoint position of the user by performing the same process as theprocess of step S73 of FIG. 12 and obtains the sight line direction ofthe user based on the viewpoint position. For example, the viewpointposition is calculated by detecting the positions of the face and eyesof the user from the in-image and the sight line direction is specifiedbased on the viewpoint position and the positions of the pupils of theuser on the in-image.

In step S104, the gaze position calculation unit 182 obtains the gazeposition based on the sight line direction supplied from the sight linedirection calculation unit 181 and the viewpoint position and suppliesthe gaze position to the control unit 52. For example, the gaze positioncalculation unit 182 obtains the gaze position on the out-image based onthe sight lie direction of the user, the viewpoint position of the user,and the position of the display unit 21.

In step S105, the control unit 52 generates control information used forfocal point adjustment, lightness adjustment, and white balanceadjustment of the out-image capturing unit 23 based on the gaze positionsupplied from the gaze position calculation unit 182.

For example, based on the gaze position, the control unit 52 generatesthe control information used for the focal point adjustment, theexposure adjustment, and the white balance adjustment by which thesubject at the gaze position is focused and the lightness and the whitebalance of the subject are optimum.

In step S106, the control unit 52 performs the adjustment of the focalpoint position, the lightness, and the white balance by controlling theout-image capturing unit 23 based on the control information. Thus, theout-image in which the subject at the gaze position of the user isfocused and the lightness and the white balance of the subject areappropriately adjusted is supplied from the out-image capturing unit 23to the signal processing unit 53. Further, the signal processing unit 53supplies the out-image supplied from the out-image capturing unit 23 tothe display unit 21 and the recording unit 54.

Here, the example in which the out-image capturing unit 23 performs thelightness adjustment and the white balance adjustment has beendescribed. However, the lightness adjustment or the white balanceadjustment may be performed by the signal processing unit 53.

In step S107, the display unit 21 displays the out-image supplied fromthe signal processing unit 53. Then, in step S108, the recording unit 54records the out-image supplied from the signal processing unit 53, andthen the photographing process ends.

As described above, the image processing device 11 specifies the gazeposition of the out-image which the user pays attention to based on thesight line direction of the user obtained from the in-image and performsthe focal point adjustment or the lightness adjustment based on the gazeposition. Accordingly, it is possible to acquire the higher-qualityout-image in which the subject which the user pays attention to isfocused and the lightness or the white balance of the subject isappropriate. Further, in this case, the focal point and the lightnessare appropriately adjusted although the user does not particularlyperform an input operation. Therefore, convenience can be improved.

Fourth Embodiment Configuration Example 4 of Signal Processing Unit

The example in which the photographing control of the out-image isperformed has been described above. However, photographing control of anin-image may be performed based on the in-image and an out-image.

Specifically, for example, as illustrated in FIG. 17, a user U21 who isa photographer is assumed to orient the out-image capturing unit 23 ofthe image processing device 11 from a room to the outside to photographa sunset landscape as an out-image P31 and photograph the user U21 as anin-image. Then, the image processing device 11 is assumed to synthesizethe in-image on the out-image P31 and acquire a synthesized imageobtained as the result as a final image which the user U21 intends toacquire.

In this case, light sources of light with which a subject is radiated atthe time of the photography differ in the inside and the outside of theroom. That is, the white balance of the out-image P31 and the whitebalance of the in-image are not identical. For this reason, when thein-image is simply synthesized with the out-image, for example, an imagein which color balance is unnatural between the user U21 on the in-imageand the background as in a synthesized image P32 may be obtained in somecases. In this example, the color of the background (landscape) of thesynthesized image P32 is dark, but the color of the user U21 is brightsince the user U21 is photographed indoors.

Accordingly, the image processing device 11 preliminarily photographsthe in-image at the time of the photography of the out-image P31 andobtains white balance values of the acquired out-image P31 and thein-image. Then, the image processing device 11 causes the display unit21 to emit light so that hues of the in-image and the out-image P31 arethe same based on the white balance values. That is, the imageprocessing device 11 obtains a light source color for adapting the whitebalance of the in-image intended to be photographed from the now to thewhite balance of the out-image P31, causes the display unit 21 to emitlight with the light source color, and re-photographs an in-image. Then,the image processing device 11 synthesizes the already photographedout-image P31 and the re-photographed in-image to generate a synthesizedimage P33.

At the time of the re-photography of the in-image, the user U21 who is asubject is radiated with appropriate light when the display unit 21emits the light. Therefore, the in-image is photographed under the samelight source as the light source of the case in which the out-image P31is photographed, and thus the hue of the user U21 on the acquiredin-image is the same as the hue of the out-image P31 which is thebackground. That is, the white balance of the out-image P31 is identicalwith the white balance of the in-image. Accordingly, the synthesizedimage P33 with the natural hue can be acquired without performing whitebalance adjustment on the photographed image.

Thus, by performing the photographing control on the in-image based onthe white balance values of the out-image and the in-image, the lightsource of the out-image is matched with the light source of the in-imageand the synthesized image with the natural hue, that is, ahigher-quality synthesized image can be acquired. The photographingcontrol is effective particularly, for example, when the outside underthe sun is photographed in the inside illuminated with a fluorescentlamp or the like and an in-image photographed in the inside and anout-image photographed outside are synthesized.

When the photographing control is performed, the signal processing unit53 of the image processing device 11 in FIG. 2 has, for example, aconfiguration illustrated in FIG. 18.

The signal processing unit 53 illustrated in FIG. 18 includes a whitebalance calculation unit 211, a white balance calculation unit 212, alight source color calculation unit 213, and a synthesis unit 214.

The white balance calculation unit 211 calculates the white balancevalue of the out-image supplied from the out-image capturing unit 23 andsupplies the white balance value to the light source color calculationunit 213. The white balance calculation unit 212 calculates the whitebalance value of the in-image supplied from the in-image capturing unit22 and supplies the white balance value to the light source colorcalculation unit 213.

The light source color calculation unit 213 calculates light sourcecolor information for adapting the white balance of the in-image to thewhite balance of the out-image based on the white balance valuessupplied from the white balance calculation unit 211 and the whitebalance calculation unit 212 and supplies the light source colorinformation to the control unit 52. The control unit 52 generatescontrol information for causing the display unit 21 to emit light basedon the light source color information from the light source colorcalculation unit 213.

The synthesis unit 214 generates a synthesized image by synthesizing theout-image from the out-image capturing unit 23 and the in-image from thein-image capturing unit 22 and supplies the synthesized image to therecording unit 54.

[Description 5 of Photographing Process]

Here, when the user operates the operation input unit 51 to select amode in which the out-image and the in-image are synthesized to generatethe synthesized image and gives an instruction to start photographing,the image processing device 11 starts the photographing process andgenerates the synthesized image. Hereinafter, the photographing processperformed when the signal processing unit 53 has the configurationillustrated in FIG. 18 will be described with reference to the flowchartof FIG. 19.

Since processes of step S131 and step S132 are the same as the processesof step S11 and step S12 of FIG. 4, the description thereof will beomitted. However, in step S131 and step S132, still images arephotographed as the out-image and the in-image. In particular, thein-image photographed in step S132 is photographed to generate the lightsource color information and is not used for the synthesis with theout-image.

The out-image is supplied from the out-image capturing unit 23 to thewhite balance calculation unit 211 and the synthesis unit 214 and thein-image is supplied from the in-image capturing unit 22 to the whitebalance calculation unit 212 and the synthesis unit 214.

In step S133, the white balance calculation unit 211 calculates thewhite balance value of the out-image supplied from the out-imagecapturing unit 23 and supplies the white balance value to the lightsource color calculation unit 213. For example, the white balancecalculation unit 211 calculates a white balance value Wr1 of an R (red)pixel of the out-image and a white balance value Wb1 of a B (blue) pixelof the out-image.

In step S134, the white balance calculation unit 212 calculates thewhite balance value of the in-image supplied from the in-image capturingunit 22 and supplies the white balance value to the light source colorcalculation unit 213. For example, the white balance calculation unit212 calculates a white balance value Wr2 of an R (red) pixel of thein-image and a white balance value Wb2 of a B (blue) pixel of thein-image.

In step S135, the light source color calculation unit 213 determineswhether the light sources of the out-image and the in-image are the samebased on the white balance values from the white balance calculationunit 211 and the white balance values from the white balance calculationunit 212.

For example, the light source color calculation unit 213 determines thatthe light sources are the same, when the white balance values of theout-image and the in-image are the same, that is, Wr1=Wr2 and Wb1=Wb2.

In step S135, when it is determined that the light sources are not thesame, that is, the light sources differ from each other, the processproceeds to step S136.

In step S136, the light source color calculation unit 213 generates thelight source color information for adapting the white balance of thein-image to the white balance of the out-image based on the whitebalance values of the out-image and the in-image and supplies the lightsource color information to the control unit 52.

For example, the light source color calculation unit 213 performscalculation of the following Expression (5) and generates light sourcecolor information indicating a relation between the respective hues ofR, G, and B when the display unit 21 is caused to emit light. That is,information indicating the relation of Expression (5) is the lightsource color information.

[Math. 5]

R:G:B=Wr1/Wr2:1:1Wb1/Wb2  (5)

In step S137, the control unit 52 generates control information forcausing the display unit 21 to emit the light with the hue indicated bythe light source color information based on the light source colorinformation supplied from the light source color calculation unit 213and performs light-emission control of the display unit 21 based on thecontrol information. That is, the control unit 52 causes the displayunit 21 to emit the light with the hue indicated by Expression (5).

When the process of step S137 is performed or it is determined in stepS135 that the light sources are the same, the in-image capturing unit 22photographs the in-image in step S138 and supplies the in-image to thewhite balance calculation unit 212 and the synthesis unit 214.

For example, when the display unit 21 is caused to emit the light instep S137, the user who is a subject is radiated with the light from thelight source under an indoor fluorescent lamp or the like through thelight emission and the light from the display unit 21 and the in-imageis consequently photographed under the light source which is identicalwith the light source of the out-image. That is, in the photographing ofstep S138, the in-image with the same white balance as that of theout-image acquired in step S131 can be acquired.

For example, when it is determined in step S135 that the light sourcesare the same, the display unit 21 does not emit the light. However, inthis case, although the display unit 21 is particularly caused not toemit the light, the out-image and the in-image are photographed underthe same light source. That is, the white balance of the in-imagephotographed in step S138 is identical with the white balance of theout-image photographed in step S131.

In step S139, the synthesis unit 214 generates the synthesized image bysynthesizing the out-image supplied from the out-image capturing unit 23and the in-image supplied from the in-image capturing unit 22 andsupplies the synthesized image to the recording unit 54. Here, theout-image acquired in step S131 and the in-image acquired in step S138are synthesized, so that the synthesized image is generated.

When the light sources of the out-image and the in-image are the same,the process of step S138 may not be performed and the out-image acquiredin step S131 and the in-image acquired in step S132 may be synthesized.

In step S140, the recording unit 54 records the synthesized imagesupplied from the synthesis unit 214, and then the photographing processends.

As described above, the image processing device 11 causes the displayunit 21 to emit the light so that the white balance of the out-image isidentical with the white balance of the in-image based on the whitebalance values of the out-image and the in-image. Then, the imageprocessing device 11 re-photographs the in-image and generates thesynthesized image by synthesizing the acquired in-image and theout-image.

Thus, by causing the display unit 21 to emit the light with theappropriate hue and adapting the white balance of the in-image to thewhite balance of the out-image, it is possible to acquire thesynthesized image with a more natural hue. That is, a higher-qualitysynthesized image can be acquired.

In the image processing device 11, the white balance of the out-imagecan be identical with the white balance of the in-image by causing thedisplay unit 21 installed on the same surface as the in-image capturingunit 22 photographing the in-image to emit light with an appropriatecolor (display an image with an appropriate color). That is, the lightsources of the out-image and the in-image can be virtually caused to beidentical. Accordingly, even when the light sources of the out-image andthe in-image differ from each other, an image photographed under thesame light source can be acquired, and thus the synthesized image with amore natural hue can be acquired.

Fifth Embodiment Outer Appearance Configuration Example of ImageProcessing Device

As described above, the example has been described in which the displayunit 21 is installed only on the front surface side of the imageprocessing device 11, as illustrated in FIG. 1. However, display unitsmay be installed on both of the front surface and the rear surface of animage processing device.

In this case, for example, the outer appearance of the image processingdevice is configured, as illustrated in FIG. 20. In FIG. 20, the samereference numerals are given to constituent elements corresponding tothe constituent elements illustrated in FIG. 1 and the descriptionthereof will be appropriately omitted.

A first surface 261 and a second surface 262 are installed in an imageprocessing device 251 illustrated in FIG. 20, for example, as indicatedby an arrow A51.

The display unit 21 and the in-image capturing unit 22 are installed onthe first surface 261. For example, an in-image photographed by thein-image capturing unit 22 is displayed on the display unit 21. Further,the out-image capturing unit 23 and a display unit 271 are installed onthe second surface 262. An out-image photographed by the out-imagecapturing unit 23 is displayed on the display unit 271. A touch panel(not illustrated) is installed to be superimposed on the display unit271, and thus an input operation can be performed by a user.

In the example indicated by the arrow A51, the first surface 261 and thesecond surface 262 are arranged on the same plane. Further, when theimage processing device 251 in the state indicated by the arrow A51 isviewed from the upper side of the drawing, as indicated by an arrow A52,a hinge mechanism 272 configured to rotate one of the first surface 261and the second surface 262 with respect to the other thereof isinstalled between the first surface 261 and the second surface 262.

Accordingly, as indicated by an arrow A53, the first surface 261 and thesecond surface 262 can be folded by the user. In the folded state of theimage processing device 251, the first surface 261 and the secondsurface 262 are superimposed so that the display unit 21 and the displayunit 271 are located mutually outside. That is, in the state indicatedby the arrow A53, the display unit 21 and the in-image capturing unit 22are located to the left side in the drawing of the first surface 261 andthe display unit 271 and the out-image capturing unit 23 are located tothe right side in the drawing of the second surface 262.

Thus, when the user operates the image processing device 251 in thefolded state of the image processing device 251, the user can perform anoperation particularly without distinguishing the front surface and therear surface from each other. For example, the user can operate thefirst surface 261 toward the user side and can also operate the secondsurface 262 toward the user side. Accordingly, an image photographed byan imaging unit oriented toward the user side is an in-image and animage photographed by an imaging unit oriented to the opposite side tothe user is an out-image.

Hereinafter, to facilitate the description, the first surface 261 isreferred to as a front surface and the second surface 262 is referred toas a rear surface. That is, the user is assumed to operate the firstsurface 261 toward the user side. Accordingly, an image photographed bythe in-image capturing unit 22 is an in-image and an image photographedby the out-image capturing unit 23 is an out-image.

Configuration Example 3 of Image Processing Device

More specifically, the image processing device 251 has a configurationillustrated in FIG. 21. In FIG. 21, the same reference numerals aregiven to constituent elements corresponding to the constituent elementsillustrated in FIG. 2 or 20 and the description thereof will beappropriately omitted.

The image processing device 251 illustrated in FIG. 21 includes adisplay unit 21, a display unit 271, an in-image capturing unit 22, anout-image capturing unit 23, an operation input unit 51, a control unit52, a signal processing unit 53, and a recording unit 54.

The image processing device 251 is configured such that the display unit271 is further included with respect to the image processing device 11illustrated in FIG. 2. The display unit 271 displays an image suppliedfrom the signal processing unit 53 under the control of the control unit52. In the display unit 271, a touch panel serving as the operationinput unit 51 is installed to be superimposed, and thus the user caninput various instructions by operating the touch panel serving as theoperation input unit 51.

[Description 6 of Photographing Process]

The image processing device 251 having such a configuration can performthe same process as the photographing process performed by theabove-described image processing device 11.

For example, when the image processing device 251 performs the sameprocess as the photographing process described with reference to FIG.19, the following process is performed. Hereinafter, a photographingprocess performed by the image processing device 251 will be describedwith reference to the flowchart of FIG. 22.

In this case, the signal processing unit 53 of the image processingdevice 251 has a configuration illustrated in FIG. 18. In thephotographing process, the white balance of an image of the userphotographed by the in-image capturing unit 22, that is, the whitebalance of the in-image is assumed to be adapted to the white balance ofthe out-image.

When the photographing process starts, processes of step S171 to stepS175 are performed. Since these processes are the same as the processesof step S131 to step S135 of FIG. 19, the description thereof will beomitted.

When it is determined in step S175 that the light sources of thein-image and the out-image are different from each other, the process ofstep S176 is performed. That is, in step S176, the light source colorcalculation unit 213 calculates the foregoing Expression (5) based onwhite balance values of the out-image and the in-image and generateslight source color information for adapting the white balance of thein-image to the white balance of the out-image.

The light source color calculation unit 213 supplies the light sourcecolor information acquired in this way to the control unit 52.

In step S177, the control unit 52 generates control information forcausing the display unit 21 to emit light with a hue indicated by thelight source color information supplied from the light source colorcalculation unit 213 and performs light-emission control of the displayunit 21 located on an inside, that is, the user side based on thecontrol information.

When the process of step S177 is performed or it is determined in stepS175 that the light sources are the same, the in-image capturing unit 22photographs an in-image of the user as a subject in step S178 andsupplies the in-image to the white balance calculation unit 212 and thesynthesis unit 214.

Thereafter, processes of step S179 and step S180 are performed, and thenthe photographing process ends. Since these processes are the same asthe processes of step S139 and step S140 of FIG. 19, the descriptionthereof will be omitted.

As described above, the image processing device 251 causes the displayunit 21 to emit light so that the white balance of the out-image and thewhite balance of the in-image are identical with each other based on thewhite balance values of the out-image and the in-image. Then, the imageprocessing device 251 re-photographs an in-image and generates asynthesized image by synthesizing the acquired in-image and theout-image.

Thus, by causing the display unit 21 to emit the light with theappropriately hue, it is possible to acquire the synthesized image witha more natural hue.

(Another Example of Photographing Process)

In the photographing process described with reference to FIG. 22, theexample has been described in which the white balance of the in-image isadapted to the white balance of the out-image. However, the whitebalance of the out-image may be adapted to the white balance of thein-image.

For example, in the image processing device 251, the display unit 271 isinstalled on the second surface 262 in which the out-image capturingunit 23 is installed. Therefore, when the display unit 271 is caused toappropriately emit light, the white balance of the out-image can beadapted to the white balance of the in-image. The photographing controlis effective particularly when a subject of the out-image is located ata position close to the image processing device 251.

Hereinafter, a photographing process performed by the image processingdevice 251 will be described with reference to the flowchart of FIG. 23when the white balance of the out-image is adapted to the white balanceof the in-image.

Since processes of step S211 to step S215 are the same as the processesof step S131 to step S135 of FIG. 19, the description thereof will beappropriately omitted.

When it is determined in step S215 that the light sources are not thesame, that is, the light sources differ from each other, the processproceeds to step S216.

In step S216, the light source color calculation unit 213 generateslight source color information for adapting the white balance of theout-image to the white balance of the in-image based on the whitebalance values of the out-image and the in-image and supplies the lightsource color information to the control unit 52.

For example, the light source color calculation unit 213 performscalculation of the following Expression (6) and generates light sourcecolor information indicating a relation between the respective hues ofR, G, and B when the display unit 271 is caused to emit light. That is,information indicating the relation of Expression (6) is the lightsource color information.

[Math. 6]

R:G:B=Wr2/Wr1:1:Wb2/Wb1  (6)

In step S217, the control unit 52 generates control information forcausing the display unit 271 to emit light with a hue indicated by thelight source color information from the light source color calculationunit 213 and performs light-emission control of the display unit 271located on the outside, that is, the opposite side to the user sidebased on the control information. That is, the control unit 52 causesthe display unit 271 to emit the light with the hue indicated byExpression (6). Accordingly, a subject of the out-image is radiated withthe light from the display unit 271.

When the process of step S217 is performed or it is determined in stepS215 that the light sources are the same, the out-image capturing unit23 photographs the out-image in step S218 and supplies the out-image tothe white balance calculation unit 211 and the synthesis unit 214.

For example, when the display unit 271 is caused to emit the light instep S217, the subject of the out-image is radiated with the light fromthe light source under an outside the sunlight or the like through thelight emission and the light from the display unit 271 and the out-imageis consequently photographed under the light source which is identicalwith the light source of the in-image. That is, in the photographing ofstep S218, the out-image with the same white balance as that of thein-image acquired in step S212 can be acquired.

For example, when it is determined in step S215 that the light sourcesare the same, the display unit 271 does not emit the light. However, inthis case, the in-image and the out-image are photographed under thesame light source.

When the out-image is re-photographed in this way, processes of stepS219 and step S220 are subsequently performed, and then thephotographing process ends. Since these processes are the same as theprocesses of step S139 and step S140 of FIG. 19, the description thereofwill be omitted. However, in step S219, the in-image acquired in stepS212 and the out-image acquired in step S218 are synthesized.

As described above, the image processing device 251 causes the displayunit 271 to emit the light so that the white balance of the in-image isidentical with the white balance of the out-image based on the whitebalance values of the out-image and the in-image. Then, the imageprocessing device 251 re-photographs the out-image and generates thesynthesized image by synthesizing the acquired out-image and theacquired in-image.

Thus, by causing the display unit 271 to emit the light with theappropriate hue and adapting the white balance of the out-image to thewhite balance of the in-image, it is possible to acquire the synthesizedimage with a more natural hue. That is, a higher-quality synthesizedimage can be acquired.

The image processing device 251 may be configured such that the user canselect whether the display unit 21 is caused to emit the light or thedisplay unit 271 is caused to emit the light. When the user can selectwhether the display unit 21 or the display unit 271 is caused to emitthe light, for example, when the display unit installed on the surfaceat which a subject is closer to the image processing device 251 iscaused to display the light, a more natural synthesized image can beacquired.

The above-described series of processes may be performed by hardware ormay be performed by software. When the series of processes is performedby software, a program of the software is installed in a computer. Here,examples of the computer include a computer embedded in dedicatedhardware or a general-purpose personal computer capable of executingvarious functions by installing various programs.

FIG. 24 is a block diagram illustrating a hardware configuration exampleof a computer executing the above-described series of processes by aprogram.

In the computer, a central processing unit (CPU) 501, a read-only memory(ROM) 502, and a random access memory (RAM) 503 are connected to eachother by a bus 504.

Further, an input/output interface 505 is connected to the bus 504. Aninput unit 506, an output unit 507, a recording unit 508, acommunication unit 509, and a drive 510 are connected to theinput/output interface 505.

The input unit 506 is configured by a keyboard, a mouse, a microphone,an imaging element, or the like. The output unit 507 is configured by adisplay, a speaker, or the like. The recording unit 508 is configured bya hard disk, a non-volatile memory, or the like. The communication unit509 is configured by a network interface or the like. The drive 510drives a removable medium 511 such as a magnetic disk, an optical disc,a magneto-optical disc, or a semiconductor memory.

In the computer having the above-described configuration, for example,the CPU 501 performs the above-described series of processes by loadingthe program recorded on the recording unit 508 to the RAM 503 via theinput/output interface 505 and the bus 504 and executing the program.

For example, the program executed by the computer (the CPU 501) can berecorded and shared in the removable medium 511 serving as a packagemedium. The program can be provided via a wired or wireless transmissionmedium such as a local area network, the Internet, or digital satellitebroadcast.

In the computer, the program can be installed on the recording unit 508via the input/output interface 505 by mounting the removable medium 511on the drive 510. The program can be received by the communication unit509 via a wired or wireless transmission medium and can be installed onthe recording unit 508. Further, the program can be installed in advancein the ROM 502 or the recording unit 508.

The program executed by the computer may be a program processedchronologically in an order described in this specification or may be aprogram processed in parallel or at a necessary timing such as a calledtime.

Embodiments of the present technology are not limited to theabove-described embodiments, but may be modified in various ways withinthe scope of the present technology without departing from the gist ofthe present technology.

For example, an embodiment of the present technology may be realized bya cloud computing configuration in which one function is distributed andprocessed collectively by a plurality of devices via a network.

The respective steps described in the above-described flowcharts may beperformed by one device and may be also distributed and performed by aplurality of devices.

When a plurality of processes are included in one step, the plurality ofprocesses included in the one step may be performed by one device or maybe distributed and performed by a plurality of devices.

Further, embodiments of the present technology may be realized asfollows.

(1) An information processing system including: circuitry configured toacquire first image data captured by a first camera unit disposed on afirst side of a housing; acquire second image data captured by a secondcamera unit disposed on a second side of the housing, which is oppositeto the first side of the housing; and modify the second image data basedon the first image data.

(2) The information processing system of (1), wherein the first imagedata is captured by the first camera unit at substantially the same timeas the second image data is captured by the second camera unit.

(3) The information processing system of any of (1) to (2), wherein thecircuitry is configured to modify the second image data by performingblur correction on the second image data based on the first image data.

(4) The information processing system of any of (1) to (3), wherein thecircuitry is configured to: generate first motion information based onthe first image data; generate second motion information based on thesecond image data; and modify the second image data by performing blurcorrection on the second image data based on the first motioninformation and the second motion information.

(5) The information processing system of (4), wherein the first motioninformation is a first motion vector corresponding to the first imagedata, and the second motion information is a second motion vectorcorresponding to the second image data.

(6) The information processing system of (5), wherein the circuitry isconfigured to: generate a third motion vector by synthesizing the firstmotion vector and the second motion vector; and modify the second imagedata by performing blur correction on the second image data based on thethird motion vector.

(7) The information processing system of claim 1, wherein the circuitryis configured to: generate first motion information corresponding tomotion of the information processing apparatus based on the first imagedata; generate second motion information corresponding to the motion ofthe information processing apparatus based on the second image data;generate third motion information corresponding to the motion of theinformation processing apparatus based on the first motion informationand the second motion information; modify the second image data byperforming blur correction on the second image data based on the thirdmotion information.

(8) The information processing system of any of (1) to (7), wherein thecircuitry is configured to modify the second image data by controllingan image capture angle corresponding to the second image data based onthe first image data.

(9) The information processing system of (8), wherein the circuitry isconfigured to control the image capture angle corresponding to thesecond image data by at least one of changing a zoom ratio of the secondcamera unit, changing a diaphragm of the second camera unit, changing anorientation of the second camera unit, and performing image deformationof the second image data.

(10) The information processing system of any of (8) to (9), wherein thecircuitry is configured to: calculate a viewpoint of a user based on thefirst image data; and control the image capture angle corresponding tothe second image data based on the calculated viewpoint of the user.

(11) The information processing system of (10), wherein the circuitry isconfigured to calculate the viewpoint of a user by obtaining a sightline direction of the user in relation to a display included in theinformation processing system.

(12) The information processing system of any of (1) to (11), whereinthe circuitry is configured to: output the second image data to adisplay; detect a gaze position of a user on the display based on thefirst image data; and modify the second image data by controlling animage capture parameter of the second camera unit based on the detectedgaze position of the user on the display.

(13) The information processing system of (12), wherein a plurality ofobjects are included in the second image data; and the circuitry isconfigured to determine which of the plurality of objects displayed onthe display corresponds to the detected gaze position of the user; andcontrol the image capture parameter corresponding to the second imagedata based on the determining.

(14) The information processing system of any of (1) to (13), whereinthe circuitry is configured modify the second image data by synthesizingthe first image data and the second image data.

(15) The information processing system of any of (1) to (14), whereinthe circuitry is configured to: obtain a first white balance valuecorresponding to the first image data; obtain a second white balancevalue corresponding to the second image data; and control a light sourceto emit light based on the first white balance value and the secondwhite balance value.

(16) The information processing system of (15), wherein the circuitry isconfigured to: acquire third image data captured by the first cameraunit while the light source is emitting light; and modify the secondimage data by synthesizing the third image data and the second imagedata.

(17) The system of any of (1) to (16), wherein the system is a mobilephone including the housing and the circuitry.

(18) The system of any of (1) to (17), wherein the system comprises: aserver including the circuitry and an interface configured to receivethe first and second image data from a device connected to the servervia a network.

(19) An information processing method including: acquiring first imagedata captured by a first camera unit disposed on a first side of ahousing; acquiring second image data captured by a second camera unitdisposed on a second side of the housing, which is opposite to the firstside of the housing; and modifying the second image data based on thefirst image data.

(20) A non-transitory computer-readable medium includingcomputer-program instructions, which when executed by an informationprocessing system, cause the system to: acquire first image datacaptured by a first camera unit disposed on a first side of a housing;acquire second image data captured by a second camera unit disposed on asecond side of the housing, which is opposite to the first side of thehousing; and modify the second image data based on the first image data.

(21) An image processing device includes: a first imaging unit thatperforms photographing in a first direction; a second imaging unit thatperforms photographing in a second direction different from the firstdirection; and a control information generation unit that generatescontrol information for adjustment of a second image obtained throughthe photographing by the second imaging unit based on informationobtained from a first image obtained through the photographing by thefirst imaging unit.

(22) In the image processing device described in (21), the controlinformation generation unit may generate motion information indicating amotion of the image processing device as the control information basedon motion information obtained from the first image. The imageprocessing device may further include a correction unit that correctsmotion shake of the second image caused by the image processing devicebased on the control information.

(23) In the image processing device described in (21) or (22), thecontrol information generation unit may generate the control informationby synthesizing the motion information obtained from the first image andmotion information obtained from the second image.

(24) In the image processing device described in any of (21) through(23), the first imaging unit may be provided on a surface of a user sideon which the image processing device is operated. The controlinformation generation unit may generate the control information forchanging a field angle of the second image based on a viewpoint positionof the user obtained from the first image.

(25) In the image processing device described in any of (21) through(24), the control information generation unit may adjust the field angleof the second image by controlling the second imaging unit based on thecontrol information.

(26) In the image processing device described in any of (21) through(25), the control information generation unit may perform field angleadjustment by changing a zoom magnification or a diaphragm of the secondimaging unit or inclining the second imaging unit.

(27) The image processing device described in any of (21) through (26)may further include an image deformation unit that adjusts the fieldangle of the second image by performing image deformation on the secondimage based on the viewpoint position.

(28) In the image processing device described in any of (21) through(27), the first imaging unit may be provided on a surface of a user sideon which the image processing device is operated. The controlinformation generation unit may generate the control information forperforming focal point adjustment of the second imaging unit, lightnessadjustment of the second image, or white balance adjustment of thesecond image based on a sight line direction of the user obtained fromthe first image.

(29) The image processing device described in any of (21) through (28)may further include a display unit that is provided on a surface of aside of the second imaging unit. The control information generation unitmay generate the control information for radiating a subject of thesecond image with light by causing the display unit to emit the lightbased on a white balance value of the first image and a white balancevalue of the second image.

(30) The image processing device described in any of (21) through (29)may further include a synthesis unit that synthesizes the first imageand the second image photographed when the display unit emits the lightbased on the control information.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

REFERENCE SIGNS LIST

-   11 Image processing device-   21 Display unit-   22 In-image capturing unit-   23 Out-image capturing unit-   53 Signal processing unit-   81 Motion estimation unit-   82 Motion estimation unit-   84 Camera blur correction unit-   141 Position calculation unit-   142 Field angle transformation processing unit-   211 White balance calculation unit-   212 White balance calculation unit-   213 Light source color calculation unit-   214 Synthesis unit

1. An image processing apparatus comprising: a housing; a communicationunit configured to communicate with an external device via a wirelessnetwork; and a circuitry configured to acquire first image data capturedby a first camera, the first camera comprising at least one lensarranged at a first side of the housing, acquire second image datacaptured by a second camera, the second camera comprising at least onelens arranged at a second side of the housing opposite to the firstside, wherein a first optical axis of the at least one lens of the firstcamera and a second optical axis of the at least one lens of the secondcamera are approximately the same, perform a photographing control basedon the first image data and the second image data, generate asynthesized image, and send an instruction to the external device viathe wireless network, the instruction is indicative of generating thesynthesized image.
 2. The image processing apparatus of claim 1, whereinthe first image data is captured by the first camera at substantiallythe same time as the second image data is captured by the second camera.3. The image processing apparatus of claim 1, wherein the photographingcontrol includes an image quality control.
 4. The image processingapparatus of claim 1, wherein the photographing control includes a whitebalance control.
 5. The image processing apparatus of claim 1, whereinthe photographing control includes a light adjustment control.
 6. Theimage processing apparatus of claim 1, wherein the photographing controlincludes an exposure control.
 7. The image processing apparatus of claim1, wherein the circuitry is further configured to modify the secondimage data by controlling an image capture angle corresponding to thesecond image data based on the first image data.
 8. The image processingapparatus of claim 1, wherein the circuitry is further configured toreceive a program from the communication unit.
 9. The image processingapparatus of claim 1, wherein the at least one lens of the first camerais arranged at a biased location of the housing.
 10. The imageprocessing apparatus of claim 1, wherein the at least one lens of thefirst camera is arranged at an upper portion of the housing.
 11. Theimage processing apparatus of claim 1, further comprising a display unitarranged at the first side of the housing.
 12. The image processingapparatus of claim 1, further comprising a display unit that is arrangedat a lower portion of the housing.
 13. The image processing apparatus ofclaim 1, wherein the circuitry is further configured to store thesynthesized image as an image.
 14. The image processing apparatus ofclaim 1, wherein the circuitry is further configured modify the secondimage data by synthesizing the first image data and the second imagedata.
 15. An image processing method comprising: acquiring, with acircuitry, first image data captured by a first camera, the first cameracomprising at least one lens arranged at a first side of a housing;acquiring second image data captured by a second camera, the secondcamera comprising at least one lens arranged at a second side of thehousing opposite to the first side, wherein a first optical axis of theat least one lens of the first camera and a second optical axis of theat least one lens of the second camera are approximately the same;performing a photographing control based on the first image data and thesecond image data; and generate a synthesized image.
 16. The imageprocessing method of claim 16, wherein the photographing controlincludes an image quality control.
 17. The image processing method ofclaim 16, wherein the at least one lens of the first camera is arrangedat a biased location of the housing.
 18. A non-transitorycomputer-readable medium including computer-program instructions, whichwhen executed by an electronic processor, cause an image processingapparatus to perform a set of operations comprising: acquiring firstimage data captured by a first camera, the first camera comprising atleast one lens arranged at a first side of a housing; acquiring secondimage data captured by a second camera, the second camera comprising atleast one lens arranged at a second side of the housing opposite to thefirst side, wherein a first optical axis of the at least one lens of thefirst camera and a second optical axis of the at least one lens of thesecond camera are approximately the same; and performing a photographingcontrol based on the first image data and the second image data; andgenerate a synthesized image.
 19. The non-transitory computer-readablemedium of claim 18, wherein the photographing control includes an imagequality control.
 20. The non-transitory computer-readable medium ofclaim 18, wherein the at least one lens of the first camera is arrangedat a biased location of the housing.